UrbanSound dataset¶
I will use a dataset called Urbansound8K for this project. The dataset contains 8732 sound excerpts (<=4s) of urban sounds in10 classes, which are:
- Air Conditioner
- Car Horn
- Children Playing
- Dog bark
- Drilling
- Engine Idling
- Gun Shot
- Jackhammer
- Siren
- Street Music
This metadata contains a unique ID for each sound excerpt along with it's given class name.
A sample of this dataset is included with the accompanying git repo and the full dataset can be access from here.
Audio sample file data overview¶
Used sound are digital audio files in .wav format.
Sound waves are digitised by sampling them at discrete intervals known as the sampling rate.
The bit depth determines how detailed the sample will be also known as the dynamic range of the signal (typically 16bit which means a sample can range from 65,536 amplitude values).
Therefore, the data we will be analysing for each sound excerpts is essentially a one dimensional array or vector of amplitude values.
Analysing audio data¶
For audio analysis, we will be using the following libraries:
1. IPython.display.Audio¶
This allows us to play audio directly in the Jupyter Notebook.
2. Librosa¶
librosa is a Python package for music and audio processing by Brian McFee and will allow us to load audio in our notebook as a numpy array for analysis and manipulation.
You may need to install librosa using pip as follows:
pip install librosa
Auditory inspection¶
I will use IPython.display.Audio to play the audio files so we can inspect aurally.
import IPython.display as ipd
ipd.Audio('../UrbanSound Dataset sample/audio/100032-3-0-0.wav')
Visual inspection¶
I will load a sample from each class and visually inspect the data for any patterns. I will use librosa to load the audio file into an array then librosa.display and matplotlib to display the waveform.
# Load imports
import IPython.display as ipd
import librosa
import librosa.display
import matplotlib.pyplot as plt
# Class: Air Conditioner
filename = '../UrbanSound Dataset sample/audio/100852-0-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Car horn
filename = '../UrbanSound Dataset sample/audio/100648-1-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Children playing
filename = '../UrbanSound Dataset sample/audio/100263-2-0-117.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Dog bark
filename = '../UrbanSound Dataset sample/audio/100032-3-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Drilling
filename = '../UrbanSound Dataset sample/audio/103199-4-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Engine Idling
filename = '../UrbanSound Dataset sample/audio/102857-5-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Gunshot
filename = '../UrbanSound Dataset sample/audio/102305-6-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Jackhammer
filename = '../UrbanSound Dataset sample/audio/103074-7-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Siren
filename = '../UrbanSound Dataset sample/audio/102853-8-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
# Class: Street music
filename = '../UrbanSound Dataset sample/audio/101848-9-0-0.wav'
plt.figure(figsize=(12,4))
data,sample_rate = librosa.load(filename)
_ = librosa.display.waveplot(data,sr=sample_rate)
ipd.Audio(filename)
Observations¶
From a visual inspection we can see that it is tricky to visualise the difference between some of the classes.
Particularly, the waveforms for reptitive sounds for air conditioner, drilling, engine idling and jackhammer are similar in shape.
Likewise the peak in the dog barking sample is simmilar in shape to the gun shot sample (albeit the samples differ in that there are two peaks for two gunshots compared to the one peak for one dog bark). Also, the car horn is similar too.
We show to similarities between the children playing and street music.
The human ear can naturally detect the difference between the harmonics, it will be interesting to see how well a deep learning model will be able to extract the necessary features to distinguish between these classes.
However, it is easy to differentiate from the waveform shape, the difference between certain classes such as dog barking and engine idling.
Dataset Metadata¶
Here we will load the UrbanSound metadata .csv file into a Panda dataframe.
import pandas as pd
metadata = pd.read_csv('../UrbanSound Dataset sample/metadata/UrbanSound8K.csv')
metadata.head()
Class distributions¶
print(metadata.class_name.value_counts())
Observations¶
Here I can see the Class labels are unbalanced. Although 7 out of the 10 classes all have exactly 1000 samples, and siren is not far off with 929, the remaining two (car_horn, gun_shot) have significantly less samples at 43% and 37% respectively.
This will be a concern and something we may need to address later on.
Audio sample file properties¶
Next I will iterate through each of the audio sample files and extract, number of audio channels, sample rate and bit-depth.
# Load various imports
import pandas as pd
import os
import librosa
import librosa.display
from helpers.wavfilehelper import WavFileHelper
wavfilehelper = WavFileHelper()
audiodata = []
for index, row in metadata.iterrows():
file_name = os.path.join(os.path.abspath('/Volumes/Untitled/ML_Data/Urban Sound/UrbanSound8K/audio/'),'fold'+str(row["fold"])+'/',str(row["slice_file_name"]))
data = wavfilehelper.read_file_properties(file_name)
audiodata.append(data)
# Convert into a Panda dataframe
audiodf = pd.DataFrame(audiodata, columns=['num_channels','sample_rate','bit_depth'])
Audio channels¶
Most of the samples have two audio channels (meaning stereo) with a few with just the one channel (mono).
The easiest option here to make them uniform will be to merge the two channels in the stero samples into one by averaging the values of the two channels.
# num of channels
print(audiodf.num_channels.value_counts(normalize=True))
Sample rate¶
There is a wide range of Sample rates that have been used across all the samples which is a concern (ranging from 96k to 8k).
This likley means that we will have to apply a sample-rate conversion technique (either up-conversion or down-conversion) so we can see an agnostic representation of their waveform which will allow us to do a fair comparison.
# sample rates
print(audiodf.sample_rate.value_counts(normalize=True))
Bit-depth¶
There is also a wide range of bit-depths. It's likely that we may need to normalise them by taking the maximum and minimum amplitude values for a given bit-depth.
# bit depth
print(audiodf.bit_depth.value_counts(normalize=True))
Other audio properties to consider¶
I may also need to consider normalising the volume levels (wave amplitude value) if this is seen to vary greatly, by either looking at the peak volume or the RMS volume.
Algorithms and Techniques¶
The proposed solution to this problem is to apply Deep Learning techniques that have proved to be highly successful in the field of image classification. First we will extract Mel-Frequency Cepstral Coefficients (MFCC) [2] from the the audio samples on a per-frame basis with a window size of a few milliseconds. The MFCC summarises the frequency distribution across the window size, so it is possible to analyse both the frequency and time characteristics of the sound. These audio representations will allow us to identify features for classification.The next step will be to train a Deep Neural Network with these data sets and make predictions. We will begin by using a simple neural network architecture, such as Multi-Layer Perceptron before experimenting with more complex architectures such as Convolutional Neural Networks.Multi-layer perceptron’s (MLP) are classed as a type of Deep Neural Network as they are composed of more than one layer of perceptrons and use non-linear activation which distinguish them from linear perceptrons. Their architecture consists of an input layer, an output layer that ultimately make a prediction about the input, and in-between the two layers there is an arbitrary number of hidden layers.These hidden layers have no direct connection with the outside world and perform the model computations. The network is fed a labelled dataset (this being a form of supervised learning) of input-output pairs and is then trained to learn a correlation between those inputs and outputs.The training process involves adjusting the weights and biases within the perceptrons in the hidden layers in order to minimise the error.The algorithm for training an MLP is known as Backpropagation. Starting with all weights in the network being randomly assigned, the inputs do a forward pass through the network and the decision of the output layer is measured against the ground truth of the labels you want to predict. Then the weights and biases are backpropagated back though the network where an optimisation method, typically Stochastic Gradient descent is used to adjust the weights so theywill move one step closer to the error minimum on the next pass. The training phase will keep on performing this cycle on the network until it the error can go no lower which is known as convergence.Convolutional Neural Networks (CNNs) build upon the architecture of MLPs but with a number of important changes. Firstly, the layers are organised into three dimensions, width, height and depth. Secondly, the nodes in one layer do not necessarily connect to all nodes in the subsequent layer, but often just a sub region of it.This allows the CNN to perform two important stages. The first being the feature extraction phase. Here a filter window slides over the input and extracts a sum of the convolution at each location which is then stored in the feature map. A pooling process is often included between CNN layers where typically the max value in each window is taken which decreases the feature map size but retains the significant data. This is important as it reduces the dimensionality of the network meaning it reduces both the training time and likelihood of overfitting. Then lastly we have the classification phase. This is where the 3D data within the network is flattened into a 1D vector to be output.For the reasons discussed, both MLPs and CNN’s typically make good classifiers, where CNN’s in particular perform very well with image classification tasks due to their feature extraction and classification parts. I believe that this will be very effective at finding patterns within the MFCC’s much like they are effective at finding patterns within images.We will use the evaluation metrics described in earlier sections to compare the performance of these solutions against the benchmark models in the next section.